Using an innovative assessment approach on a real-world group based software project

Currently, there is a lack of practical, real-world projects on Computer Science (CS) courses at Maynooth University. Generally CS undergraduate modules are composed of 24 hours of lectures and 24 hours of labs where students learn theoretical concepts in the lectures and apply their understanding to practical lab-based exercises. The problem with this approach is that students do not gain any awareness of, or learn how to solve tasks that they are likely to encounter in a real-world industrial setting; nor do they gain experience of working as part of a team even though most software development positions involve team-based work. This paper reports on a web-based development module that incorporated a real-world group based project was re-designed and delivered. The module went well; however, assessing the work fairly was found to be difficult, especially where team members contributed at considerably varying levels was a challenge. Of particular concern was that some hard-working students were penalised by other students’ poor work and lazy students were rewarded because of more hard-working students’ work. This action research project will attempt to re-address how to assess this group-based work with a cohort of students. The goal of the research is to implement an innovative assessment structure, using peer-, self-, and coassessment, for a group based real-world project, that is deemed fair and reasonable and provided a good learning environment. 1. Educational Situation & Literature Review 1.1 Background CS230 Web Information Processing is an undergraduate module on WWW technologies in our department. The advantage of this module is that it is very practical and opportunities exist to give student’s exposure to projects they might work on in an industrial setting. This is important as currently, there is a lack of practical, real-world projects on our CS degree programmes. Typically, CS modules are composed of 24 hours of lectures and 24 hours of labs where students learn theoretical concepts in the lectures and apply their understanding to practical lab-based exercises. The problem with this approach is that students do not gain any awareness of, or learn how to solve tasks that they are likely to encounter in a real-world industrial setting nor do they gain experience of working as part of a team even though most software development positions involve team-based work. Furthermore, from an employer’s perspective there is a growing concern on the quality of recent graduates and moreover on the necessity to re-train graduates as they have gained so few employment ready skills. To address this issue in part the CS230 module was re-designed to incorporate a real-world web-based group project. Students participating on this module typically are studying on five different programmes: second year students on our dedicated Computer Science and Software Engineering degree (CSSE), second year BA / BSc Multimedia students, Higher Diploma in Information Technology students and visiting international students. This blend of technical, creative, mature and international students provides welcome diversity to the module as designing web pages requires artistic flair, technical ability and awareness of different user preferences and types. 1.2 Review of Related Literature Broadly speaking there are two main goals of assessment: (1) to provide certification of academic performance and (2) to improve the quality of student learning (OECD, 2013; Goode, 2010; Sluijsmans et al., 1999). The first goal is largely achieved through summative assessment (identifying and grading what a student has learnt) whilst the latter is typically achieved by formative assessment (providing feedback of good quality information so that the learner can benefit from it) (Boud, 1998). Evaluation and assessment should align with the principles embedded in educational goals (OECD, 2013). Current graduates are often expected to work in teams, make decisions and handle responsibilities in dynamic work environments with often non-routine abstract tasks. Graduates need to be able to analyse information, problem solve, communicate and reflect on their own role and performance. Our role in higher education must be to help prepare students for taking up such positions and thus provide opportunities to facilitate the development of higher order life learning skills (Van den Bergha et al., 2006, Boud, 1990). Alternative learning methods and environments such as group based work and projectbased learning provide for such opportunities. Such approaches fall under the category of constructivist learning where learning is actively constructed by the learner and is self-regulated, goal-driven, contextual and often collaborative. O’Farrell (2002) suggests that when designing and carrying out assessment it is important that the teachers and the students are clear as to what is expected of the student, along with the marks to be awarded. Students sometimes feel that the assessment criteria are the property of the teacher. However, there is no need for secrecy as being upfront about the assessment will direct students to what is expected of them and consequently will lead to much deeper learning. According to Tucker et al. (2007) the incorporation of group based work has increased in higher education. This is driven by the commonly accepted arguments that peer learning can improve the overall quality of student learning and group work can help develop specific generic skills sought by employers such as critical enquiry, reflection ability and communication skills that are not as easily developed through more traditional approaches (Tucker et al., 2007; Boud et al., 1999). Group assessment occurs when individuals work collaboratively to produce a piece of work. The advantage of group work for the assessor is often that the burden of marking many individual pieces of work is significantly reduced (DIT, 2008). A significant concern reported by students (Tucker et al., 2007) is that group work is fairly assessed and that individual contributions are justly rewarded. Boud et al. (1999) maintains that assessment is the single most powerful influence on learning in formal courses and in a collaborative learning environment and students must perceive the assessment methods to be credible and transparent. Schemes in which there is an explicit mix of individual and group assessment for common tasks may help to alleviate this problem. Tucker et al. (2007) contends that methods such as selfassessment and peer-assessment are valid and reliable alternatives to teacher-only assessment of individual contributions to group work. Self-assessment is concerned with learners making judgments about their own learning and achievements (Falchikov and Boyd, 1989). By doing this students' take responsibility and become more actively involved in their own learning process thereby developing their own skills as reflective practitioners capable of lifelong learning. Falchikov (2000) describes peer-assessment as being concerned with the process in which groups or individuals rate the performance of their peers on instruments designed by third parties or by the students themselves. The advantages of this approach include increasing student responsibility and involvement in the process and providing students with insight on the criteria determining the quality of their own work as well. In co-assessment students and the staff collaborate in the assessment process. Both parties work together to define a mutually agreed assessment of the student's knowledge. This approach enables students to become active players in the assessment process whilst allowing staff to maintain a certain degree of control over the final assessment (Kilic, 2016; Hall, 1995). In a project-based learning study carried out by Van den Bergha et al. (2006) it was found that students believed that co-assessment allowed them to have a certain collaborative involvement in the assessment process; that it provided a happy medium between traditional and alternative modes of assessment and appreciated that it gave them the opportunity to defend or justify themselves. Boud (2000; 1997; 1990) argues that an important aspect of assessment is the ability of students to monitor their performance and make assessments of what they need to do. Learning logs or reflective journals can be used by students to reflect on their learning process (Park, 2003; Francis, 1995). Students typically use these logs to summarise how they felt about their learning experience and draw conclusions on the process. In a study by Van den Bergha et al. (2006), it was found that students perceived the use of reflective journals as one of the most effective assessment tools on a group based project. They believed that it provided the instructor with clearer insight into the internal group and gave students the opportunity to give feedback to the instructor and to justify their actions. The students considered the reflective journal to be mainly a formative instrument, but were not opposed to it being graded. Numerous studies highlight the benefits of self-, peerand co-assessment methods. Siow (2015) states that students feel that incorporating selfassessment in their assignment makes them independent learners, think and learn more, become critical thinkers, work in a structured way, and become analytical. Tucker et al. (2007) described benefits such as promoting effective teamwork, developing professional skills in self-reflection on behaviour, developing graduate attributes for working in multidisciplinary teams and lifelong learning and shifting the student's role from passive receiver to active participant. Dochy et al. (1999) identified numerous positive effects of these methods in improving the quality of learning of students: increased student confidence in the ability to perform, increased awareness of the quality of the student's own work, increased student reflections on their own behaviour and/or performance, increased student performance on assessments, increased quality of the learning output and Increased student satisfaction. In addition, Sluijsmans et al. (1999) identifies the strengths of self-, peer-, and co-assessment methods as: development of student ownership of their own learning, motivating students and facilitating active involvement, encouraging students to become more autonomous learners, development of transferable skills and showing students that their experiences are valued and their judgments are respected. Boud (1998) recommends that well-designed assessment tasks are authentic and set in a realistic context; are worthwhile learning activities in their own right; permit a holistic rather than a fragmented approach, that is, they engage students in the whole of a process; are not repetitive for either student or assessor; prompt student self-assessment; are sufficiently flexible for students to tailor them to their own needs and interests and are not likely to be interpreted by students in a way fundamentally different to those of the designer. A method to help achieve this is constructive alignment. Constructive alignment, put forward first by Biggs (1996), is the aligning of all the components in a learning system such as the learning outcomes, the teaching methods and the assessment tasks so that the learning activities should lead to the desired learning outcomes (Hurley Lawrence, 2009; Biggs 2003). The role of the instructor is to create a learning environment that supports the learning and assessment activities appropriate to achieving the desired learning outcomes. Typically, such an approach has four main steps: define the intended learning outcomes; choose teaching and learning activities that are likely to lead to the desired learning outcomes; assess students' actual learning outcomes to see how well they match what was intended and arrive at a final grade. Building upon the educational situation described in this section and the literature review on assessment, the goal of this research study is consequently to implement an innovative assessment structure, using peer-, self-, and co-assessment, for a group based real-world project, that was deemed fair and reasonable and provided a good learning environment. 2 Methodology To provide a sound theoretical grounding this project implemented an Action Research approach. 2.1 Action Research According to Carr (1986) Action Research is a form of self-reflective enquiry undertaken by participants (teachers, students or principles, for example) in social (including educational) situations in order to improve the rationality and justice of: (a) their own social or educational practices, (b) their understanding of these practices, and (c) the situations (and institutions) in which these practices are carried out. Kurt Lewin is generally considered the ‘father’ of Action Research. Lewin first coined the term ‘Action Research’ in his 1946 paper “Action Research and Minority Problems”, characterising Action Research as a comparative research on the conditions and effects of various forms of social action and research leading to social action, using a process of a spiral of steps, each of which is composed of a circle of planning, action, and fact-finding about the result of the action (Lewin, 1946). Action Research is any systematic inquiry conducted by teachers and other educational professionals in teaching/learning environments to gather information and reflect upon how their school operates, how they teach, or how well their students learn. Information is gathered with goals including effecting positive changes in the classroom and school environments, and improving student outcomes (Mills, 2003). Action Research encourages teachers to be participant researchers to gather information to share with the educational team. This information allows immediate analysis of instructional and behaviour management issues and is used to develop the next step(s) Teachers draw from research design tools that will describe what they are seeing in order to analyse and develop solutions and thereby improve their practice. Stephen Kemmis designed a scheme to model the cyclical nature of a typical action research process as shown in Figure 1. Each cycle has four steps: plan, act, observe, and reflect. Figure 1: Simple Action Research Model (taken from O’Brien, 1998) Similarly, Kurt Lewin (1948) described the process in terms of planning, fast finding, and execution. Planning starts usually with something like a general idea. The first step is to examine the idea carefully in the light of the means available. Frequently more fact finding about the situation is required. If this first period of planning is successful, two items emerge: namely, an ‘overall plan’ of how to reach the objective and secondly, a decision in regard to the first step of action. Usually this planning has also somewhat modified the original idea. The next period is devoted to executing the first step of the overall plan and this second step is followed by certain fact-findings. This reconnaissance or fact finding has four functions. First it should evaluate the action. It shows whether what has been achieved is above or below expectation. Secondly, it gives the planners a chance to learn, that is, to gather new general insight. Thirdly, this fact-finding should serve as a basis for correctly planning the next step. Finally, it serves as a basis for modifying the ‘overall plan’. The next step again is composed of a circle of planning, executing, and reconnaissance for the purpose of evaluating the results of the second step, for preparing the rational basis for planning the third step, and for perhaps modifying again the overall plan. There is general agreement on the basic steps of an action research study: (1): identify an area of focus or concern, (2): collect data for documentation, (3): analyse and interpret data, and (4): share the information with others and develop an action plan (Arhar et al., 2001; Schoen and Nolen, 2004; Stringer, 2004; Mills, 2003). As noted by Mills (2003), action research uses elements of quantitative (e.g. comparison of standard scores) and qualitative research methods. However, the literature emphasises the data collection tools of qualitative research. These include use of observation, interviews, questionnaires, checklists, rating scales, focus groups, records, videotape, audiotape, and photographs (Arhar et al., 2001; Mills, 2003; Stringer, 2004). 2.2 Module Description The continuous assessment component of the CS230 module (worth 50%) required students to develop a website to promote the study of Computer Science at third level and specifically at Maynooth University. At the first lecture the students were asked to rate their current experience of the various technologies that would be used (XHTML, JavaScript, CSS, PhP etc). From this nine groups of five to six students were created within the class. As far as possible each group had a blend of previous experience of the technologies and had at least two CSSE students, two multimedia students and one HDipIT / international student. It was felt that this mix of students was fair and should lead to interesting discussion, design and final results. Each group was asked to work together to complete a number of tasks over the duration of the module, including:  Task 1: Promote the study of Computer Science at Third Level.  Task 2: Promote Maynooth University and the CS Department.  Task 3: Develop an interactive tool to allow prospective students to learn a CS topic.  Task 4: Do something unique related to CS or to promote CS, Maynooth University or the Department.  Complete group and student learning logs for each task. In addition, students were awarded marks for formal and informal oral presentations / discussions during the project. An assessment structure that incorporated self-assessment, through reflective learning logs, peer assessment through peer-marking of presentations and co-assessment by developing marking schemes for allocating marks for the various parts of the project with the students was constructed. The format of the assessment structure is outlined next. 2.3 Implementation of co-assessment During the project the students and the lecturer worked together to co-design three marking schemes in the following order:  Marking scheme for presentation  Marking scheme for Task 1 and Task 2  Marking scheme for Task 3 and Task 4 For the first marking scheme each group was given an identical incomplete marking scheme template and asked to modify and complete the scheme during a lab session. Each group had a mentor (postgraduate student) who assisted them but did not influence the criteria or marks they came up with. Upon completion the nine schemes were collated into a final marking scheme. To this end, it was identified where groups used different wording but appeared to be describing the same criterion and this determined the number of occurrences for each criterion. Marks were allocated proportionately to the number of occurrences. A small number of criteria which were believed to be unsuitable were not included in the final scheme. The students received (1) a copy of each group’s marking scheme, (2) a spreadsheet detailing each criteria and occurrences, and (3) the final marking scheme so they could confirm that the final scheme was in fact reliably based on their work. For the second marking scheme each group prepared a scheme from scratch. Again a spreadsheet was created detailing each criteria and the number of occurrences. During one of the lectures the students were given a copy of this spreadsheet and it was displayed to the class. Discussions between the lecturer and students occurred where each criterion was discussed and collectively an appropriate mark for each was decided upon. Considerable discussion arose in the classroom with groups being asked to explain why they wanted the criterion included with other groups arguing against its inclusion at times. Where a clear decision couldn’t be reached a vote was taken and the majority decision held. Only once did the lecturer feel it necessary to overrule a decision and that was because it was felt that the students were allocating too few marks to a piece of work that had taken them a considerable length of time. The atmosphere was very relaxed and jovial and students appeared to really enjoy the negotiations. For the third marking scheme the groups again produced the schemes from scratch and the spreadsheet was created in the same manner as before. Due to time constraints the spreadsheet and a proposed scheme were posted on Moodle and feedback on them was requested. Students commented online on the proposed marking scheme and some minor modifications to the final scheme were made. 2.4 Implementation of peer-assessment At the presentations each student was given a copy of the co-designed presentation marking scheme and asked to review a certain group (each student reviewed a single group and not their own). Unfortunately, due to time constraints the marking schemes were given to the students at the start of the presentations and thus they had minimal time to review the final scheme and prepare. The output of this process was approximately 5 to 6 reviews per group. There was a requirement that each reviewer had to ask a question to a member of the group presenting that had not answered a question previously, note their answer and the rate the quality of the answer was also added. Such an approach required each presenting student to demonstrate knowledge and facilitated the development of analytical and communication skills. The same process was followed by the lecturer with each group being marked by them. The combined peer marks, lecturer marks and two colleagues’ marks formed the final mark; with the lecturers mark more heavily weighted. 2.5 Implementation of self-assessment Students completed a structured learning log upon completion of the implementation tasks (Task 1 to 4) and some of the documentation. The goal was to allow them to reflect on their own experience and on their perception of the group experience. The same learning log template was used for each task. The log can be found in Appendix A. 2.6 Action Research Cycles The action-research project was composed of a single cycle, however part of this cycle was sub-divided into two parts. The decision to use a single cycle was based upon the limited time available for the project (seven weeks). It was felt that a single cycle of a reasonable duration would lead to more reliable and authentic feedback than two or more short cycles which failed to give the students sufficient time to get used to the novel assessment structure. In addition, the sheer volume of work and the amount of survey data generated meant that by the time a change could be made it would be too rushed. The single complete assessment cycle is composed of:  self-assessment (no changes were made during the cycle)  peer-assessment (no changes were made during the cycle) The co-assessment was composed of two cycles:  Cycle 1:development of marking scheme for presentations  Cycle 2: development of marking schemes for tasks (Task 1 to Task 4). 3. Findings, results and analysis The goal of this project was to implement an innovative assessment structure, using peer-, self-, and co-assessment, for a group based realworld project, that was deemed fair and reasonable and provided a good learning environment. This section presents findings on how well this goal was satisfied with particular attention to the highlighted keywords in the above goal statement. Several instruments were used to collect data:  Survey carried out at the half-way point in project (provided in Appendix B)  Survey carried out at project completion (provided in Appendix C)  Student interviews  Peer and mentor feedback (provided in Appendix D) Results are provided in several parts based on the assessment items which were evaluated. Specifically, findings on each of the following are provided:  Feedback on co-assessment  Feedback on self-assessment  Feedback on peer assessment  General feedback on assessment structure  Feedback on group work and the learning environment 3.1 Co-assessment feedback Cycle 1 Evidence:  Half-way survey question o Did you feel the assessment scheme, built from your suggestions, was fair? (See Appendix E).  Class Discussion Cycle 2 Evidence:  End of project question o What are your thoughts on co-assessment structure? List at least one positive and one negative. Cycle 1: At the half-way point survey results indicated that 88% of students thought that the assessment scheme was fair with a further 8% indicating that it was somewhat fair, see Figure 2 for further details. Figure 2: Findings on question ‘Do you feel the assessment scheme built from you suggests, was fair?’ Sample comments given by students included (key phrases are highlighted in bold):  ‘Yeah because it took everyone’s views into consideration’.  ‘The headings which were assessed were comprised of headings from the whole class, so yes, the assessment scheme was fair’.  ‘...there was a broad range of criteria used that judge our sites fairly’.  ‘...created a room for the students to assess other students based on their performances.....some students might not want a particular group to do better than them’. Subsequent classroom discussion revealed that while the students liked the process and appreciated being involved they felt that they should get a chance to view the final marking scheme before using it as a reviewer. This would allow them to become familiar with the final scheme and formulate questions for the groups at presentations. No other changes were suggested. It was felt that no further change was necessary given the 96% of the students who replied found the assessment scheme to be predominantly fair and this was a substantial part of the goal achieved. As the students would not be involved in the marking of the next two co-designed schemes no change was needed to the subsequent cycle. However, in future it would be worth being conscious to return the marking scheme more promptly to the students where they are using it as reviewers. Cycle 2: Upon completion of the two other marking schemes students were asked ‘What are your thoughts on co-assessment? List at least one positive and one negative. A review of the comments indicated that no student declined to give a positive comment but several students provided no negative comments. In addition, the positive comments are subjectively far more encouraging and compelling than the negative ones. Sample comments are provided in Table 1 with a detailed list given in Appendix F – it is important to note that Table 1 and subsequent tables tend to show equal numbers of positive and negative comments to give the reader a sense of the types of comments received. However, feedback from this project was predominantly positive and the tables should be read with that understanding. Positives: Negatives: ‘Good that if you identify an area as important that you can get marks for that’. ‘Bad if people suggest silly stuff like jazz hands’. ‘Student input increases interest’. ‘Time taken from lectures’. ‘Good. Positive: marking scheme is fair’. ‘...takes up extra class time’. ‘It’s very good – it allows input on what we thought was important’ ‘...some people could be negative in their marks even if the group did well’. ‘You see where the marks go, and why they go for each part’. ‘May not be taken seriously’. ‘I thought it was a really good idea because it was a lot fairer way of marking’. ‘Can be biased’. ‘Good to know what is being assessed’. ‘Well, it just take time to do assessment’. ‘Involves students so they pay attention’. ‘Takes a lot of time to decide on it’. Table 1: Comments on co-assessment Finally at the end of the project students were asked if co-assessment should be maintained on the module and if more co-assessment should be introduced on the degree. Eighty percent of students felt that co-assessment should be maintained on the module but only 53% felt more should be introduced (see Figure 3). Figure 3: Findings of statements on co-assessment Students believed that whilst co-assessment suited the current module it would not fit well with more traditionally taught modules that focused on individual learning. Overall it appears that the students enjoyed and valued the co-assessment but in future better time management is needed to quickly return the schemes to students where they are to act as reviewers. 3.2 Self-assessment feedback Evidence on students’ perception of the learning logs was obtained from the following sources:  Likert-scale questions from the end of project survey:  End of project survey question: o What do you think of using learning logs? List at least one positive and one negative. The goal of the learning logs was to give students an opportunity to reflect on their role in the project and on their group work experience. Interestingly, as depicted in Figure 4 students’ ratings of the value of learning logs in terms of usefulness, helping reflection and identifying strengths and weaknesses was far greater for the group (on average 68% responded positively) than for the self (on average 56% responded positively). This is interesting as it suggests that the learning logs that were designed have more value for reflecting on the group experience than on the individual experience. Figure 4: Findings of statements about the value of Learning Logs To further explore the value of learning logs students were asked ‘What do you think of using learning logs? List at least one positive and one negative point.’ Some sample comments are provided in Table 2 (full list in Appendix G). The principal concerns with the learning logs appears to be the length of time it takes to do them, the loss of time for items perceived to be more important, and the rigid nature of the template. These issues are reasonable and need to be dealt with for future cycles of this work. The implications are discussed in detail in the next section. Positives: Negatives: ‘Helps track work’. ‘Freeform would be better...’ ‘... a good way to reflect on what you have learned through the module.’ ‘I see no point to them’. ‘I think the learning logs helped me to assess my own progress, plus the group’. ‘It takes up time that could be spent on other tasks’. ‘It’s easier to notice problems in the group’. ‘Bad thing was that learning logs were the same for every task...’. Table 2: Comments on learning logs 3.3 Peer-assessment feedback Evidence on peer-assessment was gathered from the following sources:  Half-way survey question o Did you enjoy being a reviewer at the presentations? o Would you prefer if only the lecturer assessed the work and classmates did not have any say in the marks? Shortly after the presentation students completed the half-way survey. Students were asked ‘Did you enjoy being a reviewer at the presentations?’ Seventy-one percent replied ‘yes’, 19% replied ‘maybe’ or ‘no preference’. The remaining 10% did not enjoy being a reviewer (refer to Figure 5). A full list of comments given by students to this question is provided in Appendix H, including:  ‘Yeah it was good to critically review how other teams are doing...’